Skip to content

feat(experiment): fan-out + compare + crown#240

Open
SDSLeon wants to merge 1 commit into
masterfrom
lightcode/gentle-marten-d1545bdd
Open

feat(experiment): fan-out + compare + crown#240
SDSLeon wants to merge 1 commit into
masterfrom
lightcode/gentle-marten-d1545bdd

Conversation

@SDSLeon

@SDSLeon SDSLeon commented Jun 29, 2026

Copy link
Copy Markdown
Owner

Summary

Adds experiments: fan one prompt across multiple agent/model candidates, each in its own git worktree, then compare diffs side-by-side and merge the winner — with an optional LLM-judge crown that pre-selects the best diff and a one-line rationale (user-overridable).

This is mostly orchestration glue on top of existing primitives (threads, worktrees, diffs, the one-shot prompt runner). An experiment is a thread group (candidates share groupId === experiment.id) plus an experiments registry holding the orchestration metadata.

How to use

  1. In a project draft composer, type a prompt and click the 🧪 Run as experiment button.
  2. The New Experiment modal seeds the prompt + your current agent/model, lets you add more candidates and pick a fork-from branch.
  3. Run experiment creates one worktree + thread per candidate and opens the compact experiment board — status, diff stats (+/−, files), and per-card Open / Crown / Merge winner.
  4. Crown with AI runs an LLM judge over the candidate diffs and highlights a winner with a rationale (you can override by crowning any card).
  5. Merge winner merges that branch into the base branch and archives the losers (worktrees + branches removed).

What's included

  • Contracts / IPC: experiment.ts contract, judgeExperiment IPC procedure, supervisor judge (experimentJudge.ts) built on the existing runOneShotPromptWithFallback, with robust response parsing.
  • State: experimentSlice (persisted), experimentLauncherStore, createThreadsBatch, and openExperiment/openThreadStandalone/closeExperiment view actions; new AppView kind "experiment".
  • Orchestration (experimentActions.ts): per-candidate worktree creation (sequential — avoids index.lock races), headless startThread launch, mergeExperimentWinner (via gitMergeToSource + performWorktreeRemoval), discardExperiment, crownExperiment.
  • UI: ExperimentView board + ExperimentCandidateCard, NewExperimentModal, composer trigger, sidebar experiment-group affordance.
  • Plumbing: hydration keeps live candidates attached on reload; project/view selectors and dbStorage handle the new view kind.

Design notes (informed by prior art: cmux, Vibe Kanban, uzi, Crystal)

  • Advisory crown, never auto-merge — the judge pre-selects; merge is a separate, confirmed step.
  • Anonymized candidate labels sent to the judge ("Solution A/B/C") to avoid provider/self-enhancement bias.
  • Diff noise filtering (lockfiles, minified, sourcemaps) before judging.

Testing

  • pnpm run typecheck
  • pnpm run lint (touched files) ✅
  • pnpm exec vitest run for experimentJudge, appStore, dbStorage, threadActions, and the app integration test — 99 passing ✅

Follow-ups (not in this PR)

  • Feed test/CI pass-fail signals to the judge and show them as badges.
  • Auto-approve permission prompts during fan-out so candidates don't stall.
  • i18n: user-facing strings in the new UI still need Lingui wrapping + catalog translations.

🤖 Generated with Claude Code

Introduce experiments that fan one prompt across multiple agent/model
candidates, each in its own worktree, for side-by-side diff comparison.

- add experiment contract, IPC procedure, and supervisor judge with
  one-shot prompt runner and response parsing
- add experiment store slice, launcher store, and actions to create
  candidate threads in batch and crown a winner
- add experiment app view with candidate cards and NewExperimentModal
- wire experiment view into routing, hydration, overlays, and sidebar
- persist experiments through dbStorage and project/view selectors
@vercel

vercel Bot commented Jun 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
lightcode-landing Skipped Skipped Jun 29, 2026 5:09pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant